Changed to Spool[(K, V)].
Not completely sure if my implementation is correct.

Thanks for the further feedback. Updated the diff.

@mosesn +10 for a Cursor interface.

I'm waiting for this to go to develop before I pitched the idea for a way to iterate the keys of a store, often times a less expensive operation. You can now do this in redis with the new scan flavored commands in 2.8 as a relatively cheap operation with the one requirement that you pass in what could be considered a cursor. In order to implement this I'd need the cursor to be parameterized in terms of its input so I'd want to change your suggestions to something like

trait Cursor[S, A] {
  def fetch(state: S): Future[Seq[A]]
}

or even just a function alias defined in the iterable store companion object

type Cursor[S, A] = (S => Future[Seq[A]])

I'm probably getting a head of myself by I thought it would be worth bringing up as other types of iterable ( by value or by key) stores may have the same requirements.

Thanks for the ideas. My initial thinking was that IterableStore can be used for iterating over key-values, keys, etc. and a separate Paging store can be used for paginated reads. However, now I realize Iterable is just a special case of Paging store (i.e. batch size 1).

Given this, I think a cursor makes sense and can be applied across these different use cases. To me, it seems like the fetch call would also need to return an updated cursor to be used for the subsequent fetch. Something like:

trait Cursor[S, A] {
  def fetch(state: S): Future[(S, Seq[A])]
}

(Let me know if you guys think there's a better way to capture the updated cursor.)

For stores like redis, S could be an integer representing the cursor in scan commands.
For mysql, S could be (startingRowNum, batchSize) to be applied as LIMIT to the queries.
For cache stores like MapStore etc, S can just be the backing map's iterator (or (iterator, batchSize) for the paging case).

Does this adequately cover the use cases we foresee as of now?

This was something that occurred to me when looking at the instrumentation branch. The *Store interfaces are general enough to be remixed into combinator features so that store combinators can be implemented in terms of other stores.

trait Cursor[S,A] extends ReadableStore[S, Seq[A]]

S denotes cursor state and A denotes the value ( or potentially key ) type of the store.

Want the next page?

cursor.get(state)

Want to fetch multiple pages at once?

cursor.multiGet(states)

It would actually be this, right?

trait Cursor[S,A] extends ReadableStore[S, (S,Seq[A])]

And yes, that feels right to me - it's similar to a pivot combinator...

ah yes. a client would need access to S to get the next Cursor so that makes sense.

+1

+1

@avibryant for my edification, do you have any links to more information on the pivot combinator?

I was referring to https://github.com/twitter/storehaus/blob/develop/storehaus-core/src/main/scala/com/twitter/storehaus/PivotedReadableStore.scala ... not exactly the same thing, but similar in that it is transforming the key space in a pretty significant way.

Sounds good. One remaining point.. how do we denote start state (or empty cursor) for the first get?

One way is to force Option[S] and have None denote the start state. Alternatively, we can let the implementation define it.

abstract class CursorState[S] {
  def currentState: S
  def startState: S
}

trait Cursor[S,A] extends ReadableStore[CursorState[S], (CursorState[S],Seq[A])]

I have no strong preference, but the latter seems more flexible?

Sorry for the churn. This did not occur to me earlier.

We could require a zero type class at construction time.
On Dec 4, 2013 2:26 AM, "Ruban Monu" [email protected] wrote:

Sounds good. One remaining point.. how do we denote start state (or empty
cursor) for the first get?

One way is to force Option[S] and have None denote the start state.
Alternatively, we can let the implementation define it.

abstract class CursorState[S] {
def currentState: S
def startState: S}
trait Cursor[S,A] extends ReadableStore[CursorState[S], (CursorState[S],Seq[A])]

I have no strong preference, but the latter seems more flexible?

Sorry for the churn. This did not occur to me earlier.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/191#issuecomment-29784655
.

If you wanted a split between pagable stores and iterable stores, for the pagable flavor you may not need the first type arg to be any more structured/complex than an abstract type, S. For mysql this may just be an offset/limit tuple for instance

here's a rough sketch

case class MySQLCursor[T](query: MySqlQuery[T]) extends Cursor[(Long, Long), T] {
   def get(state: (Long, Long)) = {
     val (offset, limit) = state
     query.offset(offset).limit(limit).fetch().map { values =>
         ((offset + limit, limit), values)
      }
   }
}

val cursor = MySQLCursor(mysqlQuery)
def all(from: (Long, Long)) = cursor.get(from).map {
  case (next, values) = values ++ all(next)
}
// read 100 at a time
all((0, 100))

iterable stores would need more structure like an initial state, and an end state to indicate there are no more results. A type class for this would probably be best

This looks a little like the State monad actually.

@mosesn if we pull too hard on that we will see that all of storehaus is just kleisli composition on the Future[Option[_]] monad, and these generalizations are just changing the monad. The read thing is T => M[U] where there is a Monad[M].

That said, I think there is value in making these combinators a bit more concrete for people

+1 @johnynek I think if you look hard enough everything reveals it self to be some form of monad :) The judgment of whether or not to make that explicit is up to the man behind the curtain. Sometimes the audience will give just as much applause during the show with or without that knowledge.

@johnynek lol ok

I don't think that an independent Cursor type is necessary. A "pageable" store is still iterable, you're just iterating over batches of entries. One approach that would be somewhat symmetrical to the WriteableStore.{put, multiPut} split would be:

trait IterableStore[K,V] {
  def getAll: Future[Spool[(K,V)]] =
    multiGetAll.flatMap { batchSpool =>
       batchSpool.flatMap(_.toSpool)
    }
  def multiGetAll: Future[Spool[Seq[(K,V)]]] =
    getAll.flatMap { spool =>
      spool.map(Seq(_))
    }
}

Such that the per-item Spool can be defined in terms of either batches or individual entries.

@softprops @avibryant On second thought, it might be clearer to make the trait for "Splittable" different from the trait for "Iterable"... but I still don't think an explicit "Cursor" type is necessary.

The sketch above (#191 (comment)) is sufficient for Iterable, and is easily implemented in memory by any hash map.

But there are extra considerations in the Splittable case that make it more difficult for an implementer:

1. consumers might want to execute the split method in one process, and then distribute the splits/ranges to multiple processes for execution (in which case K would need to be serializable, but we can leave that to the consumer)
2. would need to be able to generate a Store bounded by two keys/Cursors (which a JMap/Map will not let you do, but which SortedMap will)

So a potential Splittable trait might be:

trait Splittable[K,V] {
  type Range = (K,K)
  /** Generates a sequence of Ranges that together include this entire Store. */
  def split(...): Future[Seq[Range]]
  /** Returns a view of a sub-range of this Store. Analogous to Java's SortedMap.subMap method. */
  def subStore(range: Range): this.type
}

Man, so much to catch up on lately.

trait IterableStore[K,V] {
def getAll: Future[Spool[(K,V)]] =
multiGetAll.flatMap { batchSpool =>
batchSpool.flatMap(_.toSpool)
}

If there's a way to design this so that IterableStore is itself a Store I'd be happier. Maintaining a Store identity means it could have all the benefits the other ( and future ) and Store combinators provide for free. I still think you can make IterableStore the product of some combinator iterator? on a ReadableStore that wraps the store with some extra machinery. I wish I had a bit more time to sketch out a proof of concept here but I think its doable.

trait Splittable[K,V] {
type Range = (K,K)

This is starting to sound a lot like the (Un)Pivoted family of Stores. In a similar manner, the combinator for these takes something like your split function would define except that its on a per-key/value basis. I don't think Range is sufficient for all cases of getting a slab of a store, though it may in cases like mysql range queries. I'm mainly thinking of redis scan which goes back to the cursor type thing before. I'd really look back though the (Un)Pivoted catalog before adding a new type that does something very similar but doesn't produce a new Store.

As a design goal, it would be nice to think of storehaus as a small set of primitives with a small surface area interface and sets of combinators for adding behavior while producing new stores which maintain the primitives interface so that the creation of stores can easily be a small matter of composition. I understand not all square pegs fit in round holes so I can see some leeway (preferable small), but it would something nice to keep in mind. As a user of an instance of a Store, I should mainly only have to think about the get/put operations + the K/V types. I think @sritchie and @johnynek originally had that idea backed into the design. ( and their both really good designers! )

If there's a way to design this so that IterableStore is itself a Store I'd be happier.

I was thinking it was just a trait without a self-type, similar to WritableStore. But yea, I don't see any reason why it couldn't self-type Store.

I still think you can make IterableStore the product of some combinator iterator? on a ReadableStore that
wraps the store with some extra machinery.

Maybe, but I think the collection.Iterable->Spool factory function alone probably makes it sufficiently easy to implement the IterableStore trait manually.

I'm mainly thinking of redis scan which goes back to the cursor type thing before.

Yea, that's a good point. So the Cursor type does need to be a generic parameter to Splittable.

trait SplittableStore[K,V,Cursor] {
  type Range = (Cursor,Cursor)
  ...
}

I agree with @softprops that we should try and have IterableStore itself as a store.
It probably does not align with how many backing stores allow iterating over keyspace, e.g. redis has get, put, and scan.
But -- it plays really well with how our store combinators work. So +1 to that.

My 2 cents building on the ideas proposed so far..

Store approach:

trait IterableStore[S,A] extends ReadableStore[S, (S,Seq[A])] 
// S is the cursor type
// A can be K or V or (K, V) depending on what you want to iterate over

trait SplittableStore[R, K, V] extends ReadableStore[R, ReadableStore[K, V]] // or Spool[(K, V)]
// R is the range, can be a cursor pair, or something else if applicable
// get(r: R) will be analogous to subStore or subMap

Or, combinator approach:

class IterableStore[S, K, V, A](store: ReadableStore[K, V])
  (implicit inj: Injection[(K, V), A])
  extends ReadableStore[S, (S, Seq[A])]

class SplittableStore[R, K, V](store: ReadableStore[K, V])
  extends ReadableStore[R, ReadableStore[K, V]) // or Spool[(K, V)]

Argh, sorry. Dumb UI / dumb user.

@rubanm I don't think requiring a public Cursor type in IterableStore is very helpful; it:

1. makes an unnecessary implementation detail of the Store public
2. makes things more complicated in the simple case of "turn something that is collection.Iterable into an IterableStore"
3. doesn't make things significantly simpler for the implementer of a batched IterableStore. see https://gist.github.com/stuhood/8425008
4. Makes it impossible to have an IterableStore which is not iterable from a random position

... but +1 for the proposed SplittableStore.

@stuhood Hmm valid points. So we let the stores maintain cursor state. I think this also adequately covers the paging case as long as you are not looking for random reads.

For random reads (like mysql range or limit/offset queries etc), we can use a SplittableStore. Alternatively, there is a QueryableStore in the works here #205

I've updated the pull req.

Also, thanks for the Spool fix.

Thanks, looks good! Might be worthwhile to implement IterableStore on a few of storehaus' included Stores, just to make sure we're not missing anything. Or I'd be happy to do that in a separate pull?

EDIT: Whoops: missed the MapStore update in there. Looks good with the exception of dropping fillSpool.

Fixed variance and dropped fillSpool. Thanks.